Skip to content

feat(analytics): track feature enablement & usage (task-209)#1008

Merged
stephanj merged 4 commits intomasterfrom
feature/task-209-feature-usage-analytics
Apr 13, 2026
Merged

feat(analytics): track feature enablement & usage (task-209)#1008
stephanj merged 4 commits intomasterfrom
feature/task-209-feature-usage-analytics

Conversation

@stephanj
Copy link
Copy Markdown
Collaborator

Summary

  • Extends the consent-gated GA4 pipeline (task-206 / task-208 guarantees preserved) with three new event types: feature_enabled (once per IDE session per enabled feature), feature_used (one per activated feature per prompt), and feature_counts (one per session with bucketed MCP server / custom prompt / chat memory counts).
  • New AnalyticsEventBuilder with closed per-event allowlists, enum-typed params, shape/length rejection (absolute paths incl. Windows drive letters, URLs, newlines, >128 chars). AnalyticsSessionSnapshotService (APP-level @Service) guards emission with an AtomicBoolean and preflights consent before burning the one-shot. InstrumentedMcpToolProvider counts real ToolExecutor.execute() calls in stack order Approval → Instrumented → Filtered → raw, covering both standalone MCP and MCP-inside-agent via a threaded counter. Agent events fire from StreamingPromptStrategy, NonStreamingPromptExecutionService, and SubAgentRunner.
  • Disclosure copy updated in lockstep across AnalyticsConsentNotifier, GeneralSettingsComponent, and plugin.xml. Shared schema documented at docs/analytics-schema.md as the source of truth for both this repo and ../GenieBuilder. Follow-up tasks filed: task-210 here and task-197 in GenieBuilder (with the exact TRACKED_EVENTS / EVENT_LABELS / EVENT_CATEGORIES edits and the six GA4 custom dimensions to register).

Test plan

  • ./gradlew test — full suite green (~35 analytics-focused tests across AnalyticsServiceTest, AnalyticsEventBuilderTest, AnalyticsSessionSnapshotServiceTest, BucketsTest, ProviderTypeTest, InstrumentedMcpToolProviderTest, FeatureUsageTrackerTest)
  • Existing AnalyticsServiceTest (12 cases) remains green after the buildPayload refactor — behavior-preserving for prompt_executed / model_selected
  • Shape filter rejection: absolute paths (/Users/...), URLs (https://...), Windows drive letters (C:\..., D:/..., lowercase), newlines, overly long values — all unit-tested
  • One-shot snapshot guard: snapshotIfNeeded() called twice → single emission; re-arm via settingsChanged() → second emission — unit-tested without IntelliJ platform fixtures
  • Usage-only feature IDs (semantic_search, project_context_full, project_context_selected, devoxxgenie_md) rejected if passed to trackFeatureEnabled — unit-tested
  • Manual smoke test in ./gradlew runIde: toggle RAG / MCP / Agent / Web Search in Settings and verify the GA4 endpoint receives a fresh feature_enabled snapshot on apply (requires registering the 6 new GA4 custom dimensions first — see task-197 in GenieBuilder)
  • After GenieBuilder task-197 ships: confirm the three new events land in the admin panel filtered by app_name=devoxxgenie-intellij

🤖 Generated with Claude Code

stephanj and others added 3 commits April 13, 2026 16:06
Extends the consent-gated GA4 pipeline with feature-enablement snapshots
and per-prompt feature_used events. Privacy guarantees (task-206/208) are
preserved: closed per-event param allowlists, enum-typed feature_ids,
coarse bucketed counts, shape/length rejection, fire-and-forget.

Foundation
- AnalyticsEventBuilder: generic (eventName, Map) -> JSON builder with
  closed per-event allowlist, enum-value allowlist, path/URL/newline
  shape rejection, and 128-char length cap.
- FeatureId enum (closed set) with usage-only flag for rejecting
  non-snapshot-eligible features at trackFeatureEnabled.
- ProviderType enum: LOCAL/CLOUD/NONE with OPTIONAL folded into CLOUD.
- Buckets utility: standard and chat-memory ladders with boundary tests.
- AnalyticsService refactored to route all events through the builder;
  existing trackPromptExecuted/trackModelSelected behavior preserved
  (AnalyticsServiceTest still green).

Session snapshot
- DevoxxGenieSettingsChangedTopic: MessageBus topic for settings changes.
- AnalyticsSessionSnapshotService: APP-level @service, AtomicBoolean-
  guarded one-shot per IDE session. Subscribes to the topic to re-arm,
  so any settings change triggers a fresh snapshot. Narrow FeatureEventSink
  interface keeps AnalyticsService final.
- Wired from PostStartupActivity (idempotent across project opens),
  AnalyticsConsentNotifier "Keep Enabled" action, and
  GeneralSettingsComponent#apply (via the topic).

Feature usage instrumentation
- ChatMessageContext gains projectContextFullUsed, projectContextSelectedUsed,
  devoxxGenieMdUsed booleans and a final AtomicInteger mcpCallCount.
- InstrumentedMcpToolProvider: counts real ToolExecutor.execute() invocations
  inside the wrapped executor (not provideTools), so speculative tool-list
  calls and denied/filtered tools never inflate counts.
- MCPExecutionService: new optional-counter overloads preserve the
  Approval -> Instrumented -> Filtered -> raw stack order.
- FeatureUsageTracker: static facade emitting feature_used per activated
  feature; enforces "tool_call_count only meaningful for agent/mcp".
- PromptExecutionService: calls FeatureUsageTracker.emitForPrompt in the
  task.whenComplete hook after the strategy finishes.
- StreamingPromptStrategy and NonStreamingPromptExecutionService: pass
  context.getMcpCallCount() into the MCP provider chain and emit agent
  feature_used with the AgentLoopTracker's final count on success, error,
  or cancellation.

Tests (30 new + 12 existing analytics cases stay green)
- BucketsTest, ProviderTypeTest, AnalyticsEventBuilderTest,
  AnalyticsSessionSnapshotServiceTest, InstrumentedMcpToolProviderTest.

Also carries two small task-208 follow-ups that were on the branch:
Taskfile.yml gains a `version` task, and the General settings panel is
renamed to "Analytics" to match the panel's scope.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sk-209, phase 4)

Completes task-209 by setting the ChatMessageContext analytics booleans
at their assembly sites, instrumenting SemanticSearchService and
SubAgentRunner, updating the three disclosure surfaces in lockstep, and
shipping the shared GA4 schema doc.

Context set-sites (AC #25)
- ChatMessageContextUtil.setWindowContext(): flips projectContextFullUsed
  when full project is attached; processAttachedFiles() flips
  projectContextSelectedUsed when non-empty attached files are queued.
- ChatMemoryManager.buildSystemPrompt(context): mirrors the
  useDevoxxGenieMdInPrompt + file-present gate to set devoxxGenieMdUsed.

Semantic search (AC #20)
- SemanticSearchService.search() now fires FeatureUsageTracker
  .semanticSearchUsed at invocation — query text never leaves the IDE.

Sub-agent analytics (AC #23 subtlety)
- SubAgentRunner emits its own agent feature_used event in a finally
  block so success/error/cancellation all go through. Uses
  ProviderType.fromModelProvider derived from resolvedProviderName, with
  a safe fallback to ProviderType.NONE on unknown provider strings. Sub-
  agent events are separate from parent events so the parent isn't
  double-counted.

Disclosure lockstep (ACs #11, #27)
- AnalyticsConsentNotifier: new first-run bullets covering feature
  enablement snapshot + per-prompt usage, and the "never sent" list is
  extended with MCP server names/URLs/commands/tool names and custom
  prompt names/bodies.
- GeneralSettingsComponent: matching sent/not-sent bullets in the
  settings panel.
- plugin.xml: marketplace description updated to match, and the
  Settings path is now "DevoxxGenie → Analytics" (matching the
  existing panel rename).

Schema doc (AC #13)
- docs/analytics-schema.md: single source of truth for the GA4 schema
  shared with GenieBuilder — event shapes, common params, closed
  feature_id enum, provider_type mapping (including OPTIONAL → cloud),
  both bucket ladders, and a change-process checklist.

GenieBuilder follow-up (AC #14)
- Filed task-210 in this repo's backlog as the admin-UI follow-up,
  with dependency → task-209, closed enum references, and explicit
  out-of-scope rules (never display free-form MCP server or custom
  prompt values).

Tests
- FeatureUsageTrackerTest: guardrails asserting every FeatureId wire
  value matches the schema doc, the usage-only flag is set on exactly
  the four documented ids, fromWireValue round-trips, and
  semanticSearchUsed(null) is fail-silent.

Task-209 ACs are 27/28. AC #12's explicit task-208 offline-fire-and-
forget regression is covered implicitly by the existing
AnalyticsServiceTest.asyncNetworkFailureIsSilent which now runs through
the refactored AnalyticsEventBuilder path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All findings from the second review round are fixed and the full test
suite is green.

#1 MCP-inside-agent not counted
- AgentToolProviderFactory.createToolProvider now has an overload taking
  an optional AtomicInteger mcpCallCounter and threads it into
  MCPExecutionService.createRawMCPToolProvider(counter), so agent-hosted
  MCP invocations also go through InstrumentedMcpToolProvider.
- StreamingPromptStrategy and NonStreamingPromptExecutionService pass
  context.getMcpCallCount() into the new overload.

#2 RAG/WebSearch activation flags were never set on ChatMessageContext
- SearchOptionsPanel writes ragActivated/webSearchActivated to
  DevoxxGenieStateService but ChatMessageContext.ragActivated/
  webSearchActivated were always false. ChatMessageContextUtil.createContext
  now mirrors both flags from state onto the context so
  FeatureUsageTracker.emitForPrompt fires the rag and web_search_*
  events as designed.

#3 Snapshot one-shot consumed before consent
- AnalyticsSessionSnapshotService.snapshotIfNeeded now preflights
  analyticsNoticeAcknowledged + analyticsEnabled BEFORE
  compareAndSet(false, true). The first opted-in session still emits
  after "OK, Keep Enabled" is clicked because the guard never fired.

#4 Settings-change re-arm missing from RAG/MCP/WebSearch/Agent panels
- Extracted a fail-silent helper
  DevoxxGenieSettingsChangedTopic.notifySettingsChanged() and wired it
  into every relevant apply() method: GeneralSettingsComponent,
  RAGSettingsConfigurable, MCPSettingsComponent, WebSearchProviders-
  Configurable, AgentSettingsComponent. The helper swallows any
  exception (including null MessageBus in test environments) so
  settings apply() paths never crash because analytics is unreachable.

#5 Windows drive-letter absolute paths not rejected
- AnalyticsEventBuilder.rejectShape now also drops values matching the
  pattern `[A-Za-z]:[/\\].*`, plus four new unit tests covering
  backslash, forward-slash, lowercase drive letters, and the existing
  UNC path case.

#6 Semantic search usage emitted provider_type=none
- Moved the FeatureUsageTracker.semanticSearchUsed call out of
  SemanticSearchService.search and into MessageCreationService where
  the active LanguageModel is in scope. The event now reflects the
  real provider.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@stephanj stephanj force-pushed the feature/task-209-feature-usage-analytics branch from 475ec18 to 2ec0a63 Compare April 13, 2026 15:51
All 28 acceptance criteria satisfied; final summary captured. See the
three feat/fix commits on this branch for the implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@stephanj stephanj force-pushed the feature/task-209-feature-usage-analytics branch from 2ec0a63 to 7591a00 Compare April 13, 2026 15:52
@stephanj stephanj merged commit f41bcd2 into master Apr 13, 2026
5 of 6 checks passed
@stephanj stephanj deleted the feature/task-209-feature-usage-analytics branch April 13, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant